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PREFACE 


-This  paper  reports  on  some  of  the  mathematical  results  that 
the  author  obtained  while  seeking  to  refine  the  Markov  chain 
models  .used  xn  air  ASW  tactical  analyses  in  aumerous  studies 
done  at  CNA.  In  an  effort  to  take  into  account  the  waiting  time 
in  each  state  prior  to  transition,  a  non-Markov  process  was 
postulated.  Subsequent  investigation  showed  the  process  to  be 
a  reformulation  of  a  semi-Markov  process  (c.-f.,  references  (a), 

(b) ,  and  (c)). 

In  the  present  case,  the  equations  for  the  flow  resemble  a 
multi-dimensional  renewal  process.  The  behavior  of  the  system  is 
described  by  a  probability  density  which  characterizes  the  pro¬ 
cess  at  any  time  t  >  0,  given  that  the  states  of  the  process  were 
defined  at  time  t  =  0.  It  is  shown  in  the  special  steady  state 
case  that  the  probability  distribution  yields  results  which  are 
equivalent  to  those  previously  given  (references  (a),  (b) ,  (c)). 

The  author  wishes  to  thank  Drs .  J.  Tyso.i,  J.  bram,  J.  Kadane, 
and  J.  Howe  for  their  valuable  suggestions  on  certain  aspects  of 
the  problem  discussed  in  this  paper. 


INTRODUCTION 


In  dealing  with  a  stationary  Markov  process  with  discrete 


state  space 


f  s  N 


Nw' w 


we  observe  that  the  transition  from  state 


to  state  Ej  (for  i  ^  j)  does  not  depend  upon  the  amount  of 
time  the  process  has  spent  in  state  E^.  In  fact,  it  follows  as 
a  consequence  of  the  Markovian  assumption  that  if  at  any  time  the 
process  is  observed  to  be  in  state  E^,  and  in  addition,  it  is  ob¬ 
served  that  the  process  next  makes  a  transition  to  state  E^ ,  the 
random  variable  T4 j ,  denoting  the  time  the  process  spends  in  E^ 
before  moving  to  E ^ ,  is  exponentially  distributed.  It  follows 
that  the  probability  of  making  the  transition  to  Ej  during  the  inf initesimal 
interval  [T^,  T  _  +  At)  is  just  proportional  to  At,  and  does  not 
depend  on  the  amount  of  time,  T^,  already  spent  in  E^. 

Systems  arise  though,  in  which  the  probability  of  transition 
from  E^  to  Ej  does  depend  on  the  time  already  spent  in 

E..  That  is,  if  it  is  known  that  the  process  has  been  in  E.  for 
i  1 

exactly  the  last  t units  of  time,  the  probability  that  a  transition 
to  Ej  will  occur  in  the  next  At  units  of  time  depends  not  only  on 
j  and  At  but  cn  t  as  well.  We  express  this  functionally  as 

w  _  Prob/ transit ion  from  E^  to  E^  during  process  has  been  in) 

L'3  \_the  time  interval  £r,  t  +  At)  E.  throughout  [0,-r)  ? 


-  ,  w>,. 


With  a  knowledge  of  these  functions  for  all  i,j  plus  a  description 
of  the  initial  states  (at  time  t  =  0) ,  we  can  determine  explicitly 
the  condition  of  the  process  at  any  time  t. 

DEFINITION  OF  THE  STATES 

In  order  to  make  our  process  Markovian,  we  enlarge  our  state 
space.  In  essence  we  consider  the  time  spent  in 

state  as  part  of  the  characterization  of  the  state.  More  formally, 
we  proceed  as  follows.  Let  E7 ,  ...»  EN  denote  the  finite  num¬ 

ber  of  states  of  our  original  problem.  We  call  these  states 
elementary  from  now  on.  Now  define  a  new  set  of  states  as  follows. 
Each  state  is  characterized  by  an  ordered  pair  (i,v)  where  i  is 
an  integer  1  s  i  s  N,  and  t  is  a  non-negative  real  number.  Let  S 
denote  the  set  of  all  these  states.  We  call  S  the  modified  set  of 
states.  Then  for  the  process  to  be  in  the  modified  state  (i,r) 
at  time  t  means  that  the  process  is  in  the  elementary  state  at 
time  t  and  has  been  there  for  exactly  the  last  t  units  of  time. 

It  follows  from  this  that  two  modified  states  (k,r)  and  (j,\) 
are  the  same  if  and  only  if  k  =  j  and  X  =  t.  we  can  represent 
S  pictorialiy  by  the  following  diagram  consisting  of  N  half 
lines . 


3 


A  typical  sequence  of  transitions,  starting  frcm  the  elementary 
state  E, ,  at  time  t  =  0  might  look  like 


s 

3 

l 

W 

■  — — * - 1 —  f  ^ 

~2  Tl  r3 

One  may  think  of  this  process  as  operating  as  follows. 

There  is  an  "epoch  clock"  which  is  initialized  at  time  t  -  0 

upon  the  start  of  the  process,  say  in  state  E.  .  This  "epoch 

o 

clock"  keeps  advancing  in  t  as  the  process  continues.  As  the  pro- 

cess  transfers  amongst  the  elementary  states  /E.  [  ,  the  "state 

V  l)i=l 

4 


clock"  is  initialized  at  time  ~  =  0  whenever  an  elementary  state 

is  entered.  The  "state  clock"  keeps  track  of  the  time  spent, 

in  the  elementary  state  E.  until  transition,  and  is  reset  at 

1 

T  =  0  upon  entering  a  new  elementary  state,  . 

Thus,  the  sequence  of  transitions  illustrated  above  can  be 
represented  in  an  “epoch  clock"  scale  by 


I 

i 

I 
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where  ~ ^  represents  the  amount  of  time  spent  in  the  3  elementary 
state,  and  j  now  refers  to  the  order  in  which  the  elementary 


states  are  visited. 

We  observe  now  that  our  process  with  its  modified  states  is 
Markovian.  All  the  previous  history  needed  to  make  a  transition 
from  the  elementary  state  having  been  there  '  units  of  time  is 
incorporated  into  the  modified  s^ate  (i,-). 

In  general,  given  that  the  process  is  in  at  time  t, 

the  diagram  below  indicates  the  possible  transitions  the  process 


mav  make  bv  time  t  t  it. 


\l.  0) 


(2,  O:* 


(i,  -) 


(i,  7  *  :t) 


^  (j.  0} 


(K,  0) 


t  +  at 


epoch  time  t 
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Prom  equation  (1),  we  see  that  the  probability  of  transi¬ 


tion  from  (i/t)  to  (j,0)  is  given  by  w..(t)Ai.  Furthermore,  we 
assume  that  the  probability  of  transition  from  (i,T)  to 
(i,  r  +  At)  is  given  by 

N 

w..(t)At  =  1  -  Y  w..(t)At  (2) 

n  L  ID 

j=l 

jA 


THE  GENERALIZED  CHAPMAN-KOLMOGOROV  EQUATIONS  FOR  THE  SEMI -MARKOV 
PROCESS 

In  accordance  with  the  definition  of  the  set  of  modified 

states  S,  let  (X,T)t  be  a  2-dimensional  random  variable,  where 

X  is  discrete  and  T  is  continuous.  In  p?  ;ticular,  X  takes  on 

the  values  1,  2,  ....  N  and  T  is  a  non-  jgative  real  number. 

For  a  given  epoch  time  t  >0,  we  wish  to  determine  the  probability 

density  function  for  (X,T)  .  We  denote  it  by  0. (T,t)  for 

t  D 

1  s  j  £  N,  where 


£  T  <  T+At 


(3) 
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Now  consider  the  possible  (single)  transitions  during  a  time 
interval  tt,  t  +  At)  for  the  cases  (a)  t  >  o  and  (b)  t  =  0 

(a)  t  >  0:  If  at  time  t  +  At,  (X,  T).  =  (j,  t  +  At), 

t+At 

then  at  time  t  we  must  have  had  that 

(X,  T)  =  (j,  t)  (for  sufficiently  small  At). 

Thus  we  have 


0j(t  +  At,  t  +  At)  =  0  (T,  t) 


1  - 


VT)4t 


+  o(At) 


which  yields  the  partial  differential  equation 


o0.(t,  t) 

ST 


=  -  wj(T)0j(T>  t) 


(4) 


where 


Wj(T) 


(5) 


B 


(b)  t  =  0:  This  is  the  case  in  which  the  process  just 


entered  the  elementary  state  during  the 

time  interval,  [t,  t  +  At).  So  if  at  time 

t,  (X.T)^  =  (i,s)  for  some  i  and  s,  then  at 

time  t  +  At,  (X,T). ...  =  (j,0).  Thus  the 

t+At 

probability  that  (X,T)fc  has  the  value  (j,0) 
during  the  time  interval  £t,  t  +  At)  is 
given  by 

0^ (0, t) At  +  o (At) . 

On  the  other  hand  since  the  transition  to  (j,0)  can  arise 
from  any  (i,s)  /  (j,0),  we  have  for  the  probability  that 
(X.T)fc  =  (j,0)  during  [t,  t+At)  the  relation 

t+At 

f  Y  0.  (s, t)w.  . (s) Atds. 

J0  1  1J 


Equating  the  last  two  expressions  and  letting  At  -  0  we 
get  for  1  £  j  £  N  and  t  >  0 


9 


Actually  this  equation  is  incomplete.  It  provides  no  con¬ 
tribution  for  an  initial  distribution  (at  t  =  0)  for  the  process. 
So  let  0_.  denote  the  probability  that  the  process  is  initially 
in  the  elementary  state  E^;  that  is. 


0  =  Prob  j  X  =  j  )  t  =  0}  . 


It  follows  that  )  0..  =  1.  Now  let  ri(t)  be  the  function 


r\(t)  = 


1,  t  =  0 


0,  t  /  0 


The  complete  expression  for  0^  (0,t)  becomes 


0.  (0,t)  =  V  '  0.  (s,t)w.  .  (s)ds  +  0 .  TJ ( t ) 


i^j  0 


for  all  t  £  0. 


It  can  be  shown  that  the  solution  to  equation  (4)  is 


-W  (T) 

0^  (T,t)  =  H_.  (t  -  T)e  3 


for  0  <  t  s'  t ,  where 


11 


We  assume  that  P  (0)  =  0,  and  that  the  process  must  eventually 

leave  state  E..  Hence  lim  P.(t)  =  1.  Then  since  w.(t)At  is  the 
1  t—  3  3 

probability  that  the  process  is  in  E^  during  the  interval 
[r,  t  +  at),  conditioned  on  being  in  E_.  during  £o,t)#  we  find 


w  .  (t) at  = 
3 


Prob  £  T.  <  t  + 
Prob  {  T .  s  t) 


P. (T  +  At)  -  P. (t) 
_J _ 3 _ 

1  -  Pj  (t) 


If  the  distribution  function  P ^ ( t)  is  continuous,  then  we  get  upon 
letting  Ai  -  0, 


dP. (t) 

w.(T)d7  =  3 _ 

3  1  -  P. (t) 


Solving  equation  (11)  for  P^(t)  we  have 


(11) 


-W.  (t) 

P. (t)  =  1  -  e  3 


(12) 


where  W_.(t)  is  given  by  equation  (9). 
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Trr?rr- 


Now  consider  the  term  q.  .(s).  Then 

ID 

q  (s)  As  =  w.  .(s)  [l  -  P.  (s)l  As 

1  j  —  j  i 


x  =  i,  X  =  j 
s  s+As  J 


=  Prob  l  and  X  =  i  or  j 
r  J 

for  s  <  r  <  s  +  As 


X  =  i  J  X  =  i 

V  '  .  Prob)  v 


0  s  v  <  s\ 


0  ^  v  <  s 


X  =  i,  X  =  j 
s  s+As 

Prob  J  and  X  =  l  or  j 
\  r 


N 

’l 


V 


for  s  <  r  <  s  +  As, 


=  probability  that  the  process  makes  a  direct  transition 


from  to  Ej  during  the  interval  [s,  s  +  As).  So 


qss  =  f  (s)ds  (17) 

ID  13 

is  the  probability  that  the  process,  if  in  at  time  s  =  0, 

eventually  jumps  to  by  Since  we  require  that  the  process  must 

eventually  leave  Ei  for  some  other  state,  it  follows  that 

q. .  =  0.  We  note  that 
an 


14 


-W. (x) 

*  qi  \  =  /  ■  w,  -.(x)e  1  dx 

jA  jj/i 


00  /  V 

-W  (x) 

=  J  w  (x)e  dx 
0 


-  JQ  dpi(x) 


If  we  denote  by  Q  the  matrix  of  transition  probabilities  q^, 

r  iN 

for  1  s  i,  j  «;  N,  then  the  set  of  elementary  states  ^ along 
with  the  stochastic  matrix  Q  define  a  Markov  chain.  Henceforth 
we  assume  that  this  is  an  irreducible  chain. 

Now  suppose  1\  j  denotes  the  random  variable  representing  the 
time  the  process  waits  in  Ei  before  leaving  for  ( j  ¥  i) .  (Of 
course  this  assumes  prior  knowledge  of  a  transition  to  E...)  Then 
the  conditional  probability  density  for  T_,  conditioned  on  the 
process  making  a  transition  from  to  E.  is  given  by 


15 


We  make  the  following  assumption  concerning  this  density: 


the  mean  time  t. . 

iD 


defined  by 


cc 


(19) 


is  finite. 

The  following  relationship  is  evident  from  the  waiting 
time  densities  of  equation  (18).  if  t.  denotes  the  mean  waiting 
time  in  state  E^,  then  since  by  definition 

00 

!.  =  J  xdP  (x) 

0 

and  s ince 

-W.(x) 

cLP.(x)  =  )  v.  ,(x)e  dx 

j/i 


we  have 


-V  Ax) 

)  w  (x)e  dx 

— 

,3/i 


<s 


r 

I 


xqi^(x)dx 


=  \ 


<  •. 
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( 2C ) 


We  note  here  that  the  transition  probabilities  q^  and  the 
conditional  waiting  time  distributions  t^  {x)  are  exactly  the 
quantities  postulated  in  references  (c) ,  (d) ,  and  (e) .  It  follows 
from  this  that  the  transition  mechanism  given  by  equation  (1)  is 
equivalent  to  that  given  in  the  references  just  cited.  For 
a  transition  matrix  Q  =  (q^  J  and  conditional  waiting  time  den¬ 
sities  t^j(x),  there  corresponds  a  unique  w„  (x)  .  In  fact 


q.  .t. ,{x) 

v.  .(x)  =  — lJ - 

i  1  '  7 


1  -  P.(x> 


Conversely,  the  transition  function  w^  (x)  uniquely  deter¬ 
mines  the  quantities  q^  and  t^ ^  (x)  as  we  have  seen  above. 


SOLUTION  OF  THE  EQUATIONS  FOR  THE  SEMI -MARK "V  PROCESS 

Equations  (8)  and  (16)  are  sufficient  to  determine  each 
JT(t,  t) .  Standard  theorems  of  differential  equations  guarantee 
the  existence  and  uniqueness  of  H^ (t) .  One  of  these  theorems, 
the  Picard  theorem  of  successive  approx imations/  can  be  applied  to 
develop  an  algorithm  for  the  computation  of  Hj(t).* 

*  Besides  formulating  an  algorithm  for  solution,  we  introduce  the 
approximation  procedure  at  this  point  in  order  to  display  equation 
(23) (to  follow).  This  is  needed  in  a  later  section. 


17 


We  proceed  towards  the  solution  as  follows.  Let  t-0,  t’l  be 


Similarly  set 


t. . 

H(k+i)(t,)  -  j  J  H(k)(t  -  s)Q(s)ds  +«n(t1)0.  (23) 

(k) 

Thus  from  equation  (23)  if  we  know  Q(s)  and  H  (s)  at  all  points 

of  the  partition  P,  is  fully  determined  on  P.  The  bounded— 

(k) 

ness  of  Q(s)  insures  that  H  converges  (uniquely)  to  H. 

STEADY  STATE  ANALYSIS 
Define 

t 

Y.(t)  =  f  0.  (T,t)d-  +  Tj ( t ) 0 .  •  (24) 

3  \  3  3 

In  appendix  B  we  verify  that  0^(t,  t)  is  an  honest  probability 

density  for  the  joint  random  variables  (X,  T)  .  Thus  Y.(t)  is 

t  3 

the  marginal  distribution  (as  a  function  of  t)  for  the  random 
variable  X.  In  particular 

YAU)  =  Prob  ,Xt  =  jj. 

We  wish  to  study  the  behavior  of  (t)  at  equilibrium.  That 
is,  we  are  interested  in  the  limit 

Y.,  =  i;~  Y  (t) 

J  t-1®  J 


20 


if  it  exists.  Since 


t  -W  (t)  (25) 

*  (t)  =  f  H  (t-7)e  3  dT  +  r(t)0.. 

3  J0  3 

we  note  that  this  is  a  convolution  integral  and  proceed  to  apply 
the  method  of  Laplace  transforms.  Denoting  the  Laplace  trans¬ 
form  of  a  function  f(x)  by  f*(s),  where 


f*(s)  =  j*  e  SXf(x)dx 
J0 


for  real  valued  s,  s  s  0,  and  using  the  convolution  theorem  of 
Laplace  transforms,  equation  (25)  yields 


»t(s)  =  H?(s)E*(s), 


.  -W  (r) 

where  (s)  is  t.he  Laplace  transform  of  e  J 

We  now  employ  the  well  "known  Tauberian  limit  theorem  which  says 


lim  f  (x)  =  lim  sf*(s) 
X-®  S-0  + 


whenever  one  of  the  limits  exists.  Applying  this  to  equation 
(26)  we  have 


Y  =  lim  t" 
3  s-'Cf 


(s)  =  lim  sH*(s)l  I* lim  E*(s)l. 

J  L  s-CT  ^  J  [s-‘0't  J  J 
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In  order  to  compute  lim  sH.(s)  we  go  back  to  equation  (21).  Taking  Laplace 

s-d  J 

transforms  there  (of  the  vector  quantities)  we  get 


H  (  /  =  H  (s)Q  (s) 


Multiplying  by  s  and  taking  limits  as  s  -*  0+  yields 


lim  sH*(s) 

=  lim  sK*(s) 

•  -y 

lim  Q'(s) 

_  s-Or 

_  s-O' 

s-O* 

Since  each  element  of  Q  (s)  is  some  ct.(s),  where 

J 


qt.(s)  =  j  e'bl,a.  .(x)dx; 
ij  «»0  *ij'  ' 


follows  that 


lim  q  (s)  =  j  q  (x)dx  =  q 


from  equation  (17).  Thus 


lim  Q  (s)  =  Q. 

s-O* 


22 


From  the  theory  of  Markov  chains  we  know  that  there  exists  a  vector 


n 


unique  up  to  some  positive  factor  such  that 


=  ~Q 


The  vector  ~  is  just 
chain  defined  by  the 


the  limiting  or  steady  state  probabilities  for  the  Markov 
stochastic  matrix  Q.  Thus  we  must  have 


=  lim  sH  (s) 
s-*0 


(30) 


up  to  some  positive  factor. 

It  now  remains  to  evaluate  lim  E  (s).  Since 

s"*0+ 


-W  (t) 
e  J 


Then 
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00 

r  e~stdP  (t)  =  sP*(s)  -  P  (0). 
*■  o  0  J  J 


•  I  tut !  v.£  this  in  -ho  expi*ession  for  Ft(s)  (noting  that  P.(0)  =  0)  we  get 

J 


2r(s)  =  -  1  -  !*  e"s 

0  s  [  J0 


""dP.f  t) 
O' 


But  -he  iic.lt  here  as  s  -•  &  is  indeterminate.,  so  ve  use  L'  Hospital's  rqle.  This 


lim  Bf(s)  = 

S  ~U  w 


.  *  r.-**dp.(*v) 

ds  1 


s  -  0 


^  te  stdP.(t) 

VQ  0 


s  =  0 


tdP,(t) 


do  0 


Thus  tne  limit  yields  the  mean  time  snent  in  state  E.  Drier  to  transition.  If  t. 

J  *  i 

and  TT .  are  placed  in  equation  (27)  we  arc  left  with  Y.  =  rtt.  up  to  a  constant 
•,  J  0  J 


24 


factor.  Normalizing  we  have 


- sLi_  (33) 

N 

''  rr .  t . 

J  J 
j=l 


Thus  the  steady  state  probability  of  being  in  E_.  is  essentially 
the  probability  of  being  in  of  the  imbedded  Markov  chain 
given  by  Q  but  weighted  by  the  expected  waiting  times  in  each 
state.  Furthermore  we  note  the  independence  of  y.  with  respect 
to  the  initial  condition  0.  It  is  also  evident  that  y  is  inde¬ 
pendent  of  any  of  the  specific  waiting  time  distributions.  We 
need  only  know  the  mean  waiting  time  t^. 

We  can  also  calculate  the  steady  state  value  of  0^(T,t), 
that  is. 


25 


By  equation  (8)  we  get 


-W.(T) 

0.(r)  =  lim  K.(t  -  T)e 
3  t-*® 


(up  to  a  constant  factor).  Upon  normalizing  we  get 


*«(*) 


-W,(t) 

n.e  - 

j _ 

IT  « 


j=l 


“  .t . 
J  J 


(34) 


Thus  at  steady  state  the  probability  of  being  in  for  exactly  the 
last  -  units  of  time  is  just  the  equilibrium  probability  n ^  of 
being  in  E ^  of  the  imbedded  Markov  chain  times  the  probability 
of  remaining  in  E ^  for  at  least  t  units  of  time. 

V7e  note  here  that  since 


-W,(t) 
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then 


¥,  =  f  0,(T)dT 
J  V  Q  J 


as  we  should  expect. 


FIRST  PASSAGE  AND  RECURRENCE  DISTRIBUTIONS 

Let  r^ . (t)  be  the  probability  density  function  for  first 
passage  measured  from  the  moment  the  process  enters  E^  until  it 
first  enters  E_. .  In  particular  then,  r^  (t)  is  the  probability 
density  for  the  recurrence  time  of  state  E^.  We  will  show  that 


■ — 

ri.j^  ~  L  '*:k  J  "  s^ds  +  ^(t). 


Indeed,  there  are  two  ways  that  the  process  can  first  arrive  at 


Ej  at  time  t,  having  entered  E.  at  t  =  0.  Either  the  process  re¬ 
mained  in  E.  for  a  time  s,  and  left  for  Ev  whence  it  took  t  -  s 
units  of  time  to  first  get  to  E^  or  the  process  remained  in  E^ 
for  the  whole  time  t  at  which  point  it  jumped  to  E^.  The  integral 
in  equation  (35)  reflects  the  former  possibility.  Since  the  condi¬ 
tional  waiting  time  in  E^  before  transfer  to  E^  is  independent  of 
the  first  passage  time  from  E^_  to  E^  tne  integral  in  equation  (35) 
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is  the  probability  density  function  for  the  time  t  of  first 
passage  via  the  route  E^  to  E^  to  E.. .  Since  we  can  make  all  of 
these  transitions  (through  E^)  we  must  weight  them  according  to 
their  eventual  likelihood  of  occurrence,  namely  the  q^_.‘s.  The 
latter  possibiJity  foi  first  passage  to  E^  at  time  t  is  represented 
by  the  unconditional  probability  (t)  . 

The  expected  first  passage  time  r^  can  be  shown  to  satisfy 
the  following  equation  for  all  1  <  i,  j  s  N. 


h  '  l  WTik  -  V  *  qtJ  ty 


An  exact  derivation  of  equation  (36)  is  presented  in  appendix  A. 
At  this  point  we  give  a  heuristic  proof  of  equation  (36). 

Suppose  the  process  enters  E^  at  time  t  =  0.  At  this  point 
a  choice  is  made  as  to  what  elementary  state  the  process  jumps 
to  next.  Suppose  the  successor  is  E^  where  k  ^  j.  Then  the  pro¬ 
cess  waits  in  for  an  expected  time  t.^_  before  going  to  E^. 
Since  the  waiting  time  in  E^  is  independent  of  the  first  pas¬ 
sage  time  from  E^_  to  E ^ ,  the  expected  time  from  entry  to  E^  until 

first  entry  to  E^  is  t.^  +  rk  j  *  Bufc  c*1°^ce  ^  deter- 

mir  ed  by  the  probabilities  q^.  So  we  weight  the  times  t^  +  r^ 
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accordingly  as  in  the  first  term  on  the  r.h.s.  of  equation  (38). 
The  last  term  of  equation  (35)  results  from  the  choice  to  wait 
in  E^  and  go  to  £\  directly  without  passing  through  any  inter¬ 
vening  elementary  states  E^,  This  waiting  time,  t_  is  also 
weighed  accordingly. 

Substituting  t^  for  ^  we  have  from  equation  (36) 

,  k^.i 

that 


rij  L  qikrkj  +  V 


We  are  primarily  interested  in  obtaining  the  expected  recur¬ 
rence  time  r^i  for  1  s  i  s  N.  This  is  simplified  by  putting 
equation  (37)  in  matrix  notation.  Letting  R  denote  the  matrix 
of  terms  .  and  T  the  diagonal  matrix  of  the  t^'s,  we  get 


K  »  Q(R  -  Bp)  +  T 


where  Rp  is  obtained  from  R  by  replacing  the  off  diagonal  elements 
by  zero.  Letting  RQ  =  R  -  R^  we  get 


P  =  U  -  I)B0  *  T 
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APPENDIX  A 


DERIVATION  OF  THE  EXPECTED  FIRST  PASSAGE  TIMES 


Ry  definition. 


CO 

j  _ 


Then  from  equation  (37) 


tik(s)rkj^t  “  s^ds 


dt  + 


to.  .(t)dt 
-ijv  ' 


CO 


t;k(s) 


trkJ(t 


as  + 


Qij 


=  L  V(s) 

Wo 


(s  +  x)r 


ds  + 


q. .t,  . 
tj  ij 


(where  we  have  made  the  change  of  variables  x  =  t  -  3) 


-c» 


} 

1 

k/j 


^Lk 


lk<! 


Is  +  r«J 


ds  + 


*  ^ 
kf  j 


ds  + 


a.  . 
m.J 


l.  . 

J  J 


=  '  Q.j,  t.,  -5-  r,  .  +  q.  .t. 

~  ik  L  i*  kj  J  4ij  i j 

kfj 


us  ins:  equation  (19)  to  evaluate  t.,  . 

ik 


A- 2 


**<****'I»UM^^ 


APPENDIX  B 


It  then  follows  from  the  uniform  convergence  of  H  to 


H  on  (0,t)  that  H(t)  2  0.  Since  each  (t)  2  0  then 


-w  (t) 

0,(T,  t)  =  H  (t  -  T)e  J  ^  0. 
J  J 


For  (ii)  we  define  the  function 


Y .  (t) 
D 


t 

f  0. (r,t)dT  +  n(t)0. . 
J  Q  J  D 


Then  substitution  for  0..  (r,t)  from  equation  (8)  and  (10),  we  get 


v)f  ?-  cV  'j 

|  2,  j  vK:j(s)ds  +  5(v)0j|dv  + 

r  ° 


r.V 

1  0  (s,  v)w 

0 


ij(s)ds} 


dv  +  T|  ( t)0 . . 


B-2 


Summing  over  all  states  j  we  have 


N 


N 


fl 


Y*(t)  = 
J 


j=l 


/ 

•j=l 


i  0,(s,  t)w  (s)ds  +  > 

V,  J  J 

j-1 


05(s,  t)w^(s)ds 


N 


=  -  \ 


*J(s'  t} 


k<i 


Wjk(s)ds  + 


j— 1  if  j 


0 


y/ . ( s ?  t.)w..(s)ds 


i: 


n 


5_1  \-J  -• 

J-j.  Ivj-y 


0.(s,  t)v„(s)ds  +  .  0,(s,  t)v  ,(s)ds 

J  J1'-  *•  o  - 

J=1  if.: 


=  o. 


Observe  that 


hence 


II 


j=l 


for  aj.1  t  >  0.  But  this  is  just  the  statement  (ii)  v:e  wished 


to  prove. 
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